Research in Computing Science, Vol. 65, pp. 11-23, 2013.
Abstract: We propose a fully automatic technique for evaluating text summaries without the need to prepare the gold standard summaries manually. A standard and popular summary evaluation techniques or tools are not fully automatic; they all need some manual process or manual reference summary. Using recognizing textual entailment (TE), automatically generated summaries can be evaluated completely automatically without any manual preparation process. We use a TE system based on a combination of lexical entailment module, lexical distance module, Chunk module, Named Entity module and syntactic text entailment (TE) module. The documents are used as text (T) and summary of these documents are taken as hypothesis (H). Therefore, the more information of the document is entailed by its summary the better the summary. Comparing with the ROUGH 1.5.5 evaluation scores over TAC 2008 (formerly DUC, conducted by NIST) dataset, the proposed evaluation technique predicts the ROUGH scores with a accuracy of 98.25% with respect to ROUGH-2 and 95.65% with respect to ROUGH-SU4.
Keywords: Automatic text summarization, summary evaluation, recognizing textual entailment
PDF: Entailment-based Fully Automatic Technique for Evaluation of Summaries
PDF: Entailment-based Fully Automatic Technique for Evaluation of Summaries